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Abstract 

We investigated the use of the Bayesian inference to restore noise-degraded images under con- 
ditions of spatially correlated noise. The generative statistical models used for the original image 
and the noise were assumed to obey multi-dimensional Gaussian distributions whose covariance 
matrices are translational invariant. We derived an exact description to be used as the expec- 
tation for the restored image by the Fourier transformation and restored an image distorted by 
spatially correlated noise by using a spatially uncorrelated noise model. We found that the result- 
ing hyperparameter estimations for the minimum error and maximal posterior marginal criteria 
did not coincide when the generative probabilistic model and the model used for restoration were 
in different classes, while they did coincide when they were in the same class. 
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I. INTRODUCTION 



Recent research has shown that the Bayesian inference is a useful approach to the im- 
age restoration problem [|IJ]-||. However, previous research has usually assumed that the 
superimposed noise is not correlated between pixels [[]]]- |L2]]. Statistical mechanics provides 
a useful method for analyzing the image restoration problem [f|,||. Still, when we take 
optical effects into consideration, it seems natural to take spatially correlated noise into 
account. We have investigated image restoration under the condition of spatially correlated 
noise. We considered a case in which both the original image and noise obey a Gaussian 
distribution composed of translational-symmetric matrices. We were able to diagonalize the 
Gaussian distribution by using Fourier transformation. While previous research was done 
using spatially uncorrelated noise, we considered the availability of a spatially uncorrelated 
noise model against spatially correlated noise. 

We considered tow methods for estimating the hyperparameters. The first is minimizing 
the mean squared error - the hyperparameters are set so that the mean squared error is 
minimized. However, minimizing the mean squared error is not applicable to the image 
restoration problem because we need the original image to practice this method. The second 
method is maximizing the marginal likelihood |J - the hyperparameters are estimated by 
maximizing the marginal likelihood acquired from only the distorted image. When the noise 
probabilities for generating and restoring coincidence are the same, the results given by the 
two methods are the same. When the probabilities are different, we found that they are not. 

II. MODEL 

The images we see in our daily life are always two-dimensional, although we assume a 
(i-dimensional image for mathematical generalization, where the length of each side is L and 
the total number of pixels, N, is L d . The i-th pixel value, of an original image can be 
written in a multiple Gaussian distribution as 

P(0 = 7 )n t \ exp [-#(£, (3, h)] , (1) 
H(tP,h) = f(PG + hI)t (2) 
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where (3 and h are positive scalar values, G is a matrix, and / is the unit matrix. The 
partition function, Z pTioi ((3, h), is given by 

Z prim (P,h) = (2tt)t \2(J3G+hI)\-* . (3) 

Note that Eq. (0) is called the Hamiltonian in statistical mechanics. In this paper, we argue 
that matrix G is translational invariant and that element G^ j is given by || 

G i,j = 2d5 (i-j),0 ~ Yl 6 (i-j),-S ~ Yl 6 (i-j),$> ^ 

S S 

where the d-dimensional vector S is 



6 = (1,0,...,0),(0,1,0,...,0),...,(0 J ...,0,1). (5) 
The 5^ j in Eq. (^) is the Kronecker delta, which is assumed to be 

j ; ' (6) 

Equation means that interactions occur only between nearest-neighbor pixels. If /3 takes 
a large value, i.e., the interaction of the first term has a large effect, the neighboring pixels 
tend to take the same value. When h is large, the absolute value of the pixels tends to be 
small. Furthermore, we use a periodic boundary condition, 



U = €i+Ld- ( 7 ) 

The distorted image, r = {t^}, is generated according to the following conditional proba- 
bility, which obeys a Gaussian distribution: 



1 



exp 



(8) 



Z noisc ( J R) = (2 7 r)T| J R|2, (9) 

where R is a translational invariant covariance matrix. In this paper, we consider the 
following spatially correlated noise: 



R. 



a )b 2 $i j + ab 2 exp 



(*- 3, 



(10) 
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< a < 1. The noise obeys an identical independent Gaussian distribution with mean 
and variance b 2 , when a = 0. 

A general strategy commonly used in image restoration is to apply the Bayes formulation 
to the posterior probability Here, er is a restoration image based on the Bayes formulation. 
When a distorted image, r, is given, one can use the formulation to calculate the restored 
image, er: 

P( tT \ T )= P out{r\(T)P((T) _ exp[-H eff ] 
1 1 ' J dcrP out (r|<r)P(tr) f daexp[-H eff ]' 1 ' 

where 

H eff = <r T ({3G + hl)tr + ~(t - <t) t R-\t - tr), (12) 
is an effective Hamiltonian. 



III. THEORY 

A. Expectation of the restored image 

Since the covariance matrices given by Eqs. (f4|) and fllQ|) are translational invariant, they 
can be diagonalized using the discrete Fourier transformation. 
The discrete Fourier transformation is 

3 

and the inverse Fourier transformation is 

k 

where i is the imaginary unit, and k is a d- dimensional vector with the same as j, and the 
degree of freedom is L d . 

J2 = L d = N, j2 = Ld = N - ( 15 ) 

3 k 

Moreover, each component of k takes the value 

2 4 2(L-1) 

0,-7T,-7V , V L \ . (16) 



We can diagonalize Eq. (|l|) by using the Fourier representation: 

1 



^prior(/3, h) 



exp 



5><3 fc + h)i k i_ k 
k 



% = E[ 2 - 2cos ( fc - <5 )]- 



Furthermore, we diagonalize a in Eq. (JT0|) : 



I 2 



R k = (l- a)b 2 + ab 2 ^e ~& cos(fc ■ Z), 

Z 

where the range of /, which is a component of vector Z, is given by 



(17) 



(18) 



(19) 



-(L - 1) < I < L- 1. (20) 

We can execute the Fourier transformation on the inside of exp in Eq. @ in the same way 
as in Eq. (fj). 



1 



x exp 



k 



(21) 



We define (•) as denoting the thermal average based on the Bayes formula for the posterior 



probability P(ct\t) in Eq. (jTTf) . The expectation of the restored image, aj, is thus assumed 
to be 



k 



ikj 



(22) 



Using Eq. (|TT|) , the expectation of the Fourier component, au, can be written as 



k' )Y\k' da k ie 



H. 



ft 



(23) 



where H e ff is obtained by applying the Fourier transformation to Eq. ( |12]) as follows. 

H*ff = Y^^ k + h)v k v_ k 

k 

+ Yl —( f k - °k)( f -k - °~k)- 



(24) 



k 2R k 
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(3, h, and are unknown adjustable parameters that determine the properties of the origi- 
nal image and noise. One of our major focus here is how to estimate these hyperparameters 
precisely. 

To proceed further, we have to assume some explicit form of the source prior and noise 
posterior formations, which are used when we restore the image. In this paper, we define 
both of these probability formations as the same formation except that they have different 
hyperparameters. We define the hyperparameters that correspond to f3, h„ and R k as /§, h, 
and Rfc, respectively. If R k is independent of k, the noise is spatially uncorrelated. 

Substituting 

A k = $G k + h+-^, 4 = TIT ( 25 ) 

into Eqs. (|23f) and (^4|), we get 



fa> = ^k- (26) 
A k 



Consequently, ( aj ) in Eq. (^) is given by 



1 T ^ COS [ fc - (j '- i)] 
3/ ^^k i Wk + h' 1 ' ^ 



m k 



In this paper, we regard Eq. (|27Q as representing the restored image. 



B. Minimization of mean squared error 



In this subsection, we estimate hyperparameters /?, h, and R k by using the criterion 
defined by Eq. (^7^ for minimizing the mean squared error between the original and restored 
images. The expectation of this mean squared error, Ei, is represented by 



(28) 



where ||-|| denotes the data average of a simultaneous distribution 

P(T,Z)=P out (T\t)P.(t). 

Applying the Fourier transformation to Eq. (|28|) , we can derive the next representation. 



* = ^T,\\(tk-(*k))(L k 
k 



(29) 



Since P(r, £) is diagonalized by the Fourier transformation, we can easily calculate each k 
as follows, 



d^, k df k 




x exp 



? B k~ 
Zk - ^k 



A 



k 



A 



k 



B 



k~ 



B 

\B k -^ 



k 
v k. 



\ T k\ 



' + l B k B k 



2A 



k 



where A k , B k , and Z substitute for 



k 



A k = j3G k + h + 



2R 



B 



k 



k 



2R 



k 



(30) 



(31) 
(32) 



(33) 



r fc exp 



-A 



k 



B 



k~ 



tk ~ lT f k 



l k 



B 



A 

[ k. 



\ r k\ 



Therefore, the mean squared error is represented by 

1 



E 1 = —Y 

1 2N 

k 



f3G k + h + ^ 



k 



( 



+ 



2R 



k 



2R 



k 



( 3G k + h + wr_ PG k + h' 



2R 



kJ 



x 



k 



When the conditions 



4 \P G h + h 



f3 — (5, h = h, and R k = R k , 
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(34) 



(35) 



(36) 



hold, the mean squared error takes the following minimum: 



1 



E 



1 



(37) 



Ex 



2N 



(3G k + h + 



k 



2R 



k J 



More precisely, when the ratio ft : h : Rfc is equal to the ratio (3 : h : Rfc, E\ is the minimum 
given by Eq. fl37|). Note that this equation represents the limit of the restoration. 



Likewise, we can derive the mean squared error, E2, between the original image and the 
distorted image as being represented by 



E2 depends only on the diagonal element of the noise covariance matrix. 
C. Maximization of the marginal likelihood 

The minimized mean squared error criterion generally cannot be used since the unknown 
original image itself is needed to evaluate the squared error. If we already had the original 
image, there would be no need to restore the distorted image. Hence, in this paper, we argue 
for using the maximization of the marginal likelihood. 

As shown in Eqs. ([!]) and (D), we already know the source prior and noise probabilities. 
Using both probabilities, we can derive -P(r): 




(38) 



(39) 




(40) 



P{t) is called the marginal likelihood. 

Maximization of the marginal likelihood is used to set the hyperparameters, ft, h, and R, 
to use to obtain the maximum value of P(t) for distorted image r. 

With Eqs. (§) and the marginal likelihood is given by 




(41) 
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where 



ZpoBtariarCA h,R) = Y[j dS k exp(-4//) (42) 

(43) 



k 



JV 
7T 2 



k yA k 



Because In is a monotonically increasing function that maximizes the logarithmic marginal 
likelihood, maximizing lnP(r) is equivalent to maximizing P(t). 
The logarithmic marginal likelihood In P(t) can be written as, 



In (P(t)) = In 



^posterior ft> 3 -R) 
•^noisc 



-i^ln( j §G fc + ft+4-] - y ln(27r) 

-^E ln (4)+^E ln (^fc + ^) 
+ &) 3- 

-E- — ^l-fcl 2 - (44) 

k K 2R^ 



IV. RESULT 



In this section, we consider a method to restore an image distorted by spatially correlated 
noise by means of the spatially uncorrelated noise model. To do this, we replace R as one 
of the hyperparameters: 

***j =fS ij> Rk= f - (45) 

We can now write Eq. fl27l) as 

1 ^^r^cosffe- (j -*)] 



fc 2 



Figure [l] shows the analytical and simulated mean squared errors between the original 
image and each restored image, Fig. shows the estimated f, and Fig. |3] show the estimated 
parameters h and (3. We used 100 artificially generated two-dimensional iV = 16 2 images 
with hyperparameters (3 = 1.0, h = 1.0, b = 0.75, and k = 3.0. The horizontal axes of these 



9 



0.6 



2 

CD 0.4 

"O 

CD 

CO 



0.3 - 



cc 

£ 0.1 



0.0 



















my 






^y ip^y 

^^--^ (B) 









0.0 0.2 0.4 0.6 0.8 

a from Eq. (10) 



1.0 



FIG. 1: Mean squared error between original and each restored image. Horizontal axis denotes 
a from Eq. (|io|); vertical axis denotes mean squared error. (A) Optimum decode given by Eq. 
(|37|). (B) Restored using criterion of minimum restored error. (C) Restored using f estimated 
by maximizing marginal likelihood. (D) Restored using /3, h, and r, all estimated by maximizing 
marginal likelihood. (E) Error before decoding given by Eq. (j39|). 
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FIG. 2: Estimated f. Horizontal axis denotes a from Eq. (0); vertical axis denotes average 
estimated f obtained by maximizing marginal likelihood or optimal r obtained using criterion of 
minimum restored error. Solid line (B) denotes f used in Fig. |l](B), dashed line (C) denotes f used 
in Fig. |l](C), and dotted line (D) denotes r used in Fig. |l](D). 

three figures denote a from Eq. (pi]); a represents the magnitude of the spatial correlation 
of the noise (a = implies no spatial correlation). 



10 



'&.ff=i.6r~^-— 

h 
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a from Eq. (10) 

FIG. 3: Estimated h (dotted line) and (3 (dashed line). Both values were averaged with respect 
to 100 artificially generated images. Horizontal axis denotes a from Eq. (TI^); vertical axis denotes 
average estimated h or (3 or actual h or (3. Solid line denotes actual h(= (3 = 1.0). 

A. Minimization of mean squared error 

Curve (E) in Fig. [I] represents error Ei given by Eq. (|39"1), that is, the mean squared 
error between the original and distorted images. (The dashed line represents the analytical 
error, and the solid line represents the simulated results.) Because the numbers of pixels 
(N = 16 2 ) and samples (100) were finite, the simulated results have some fluctuation. This 
fluctuation tends to asymptotically disappear as the numbers are increased. 

Curve (A) in Fig. p] represents minimum error Ei min given by Eq. (j37|), that is, the 
mean squared error between the original and distorted images restored using the spatially 
correlated noise model with the best parameters. Hence, Ei min implies the limits of image 
restoration-no further error reduction is possible. Curve (B) in Fig. [l| represents the error 
in an image restored using the uncorrelated noise model. The restoration was done using 
hyperparameter f from Eq. (|28|) , which was acquired minimizing the mean squared error. 
(The dashed line represents the analytical results, and the solid line represents the simulated 
results.) Curve (B) in Fig. |2| shows the optimal values of f by minimizing the mean squared 
error. 

Note that when a = 0.0, curve (A) in Fig. [I] is equal to curve (B). Curve (B) in Fig. [I] 
shows the optimum performance when using the minimization of the mean squared error as 
the criterion. However, as mentioned, the hyperparameters cannot be estimated in practice 
by minimizing the mean squared error. Therefore, we will discuss how the error can be 
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reduced by maximizing the logarithmic marginal likelihood in the next subsection. 



B. Maximization of the marginal likelihood 



Substituting Eq. (g4|) for Eq. (pj) , we get 

k 

-y ln(f) + i^ln( / 3G fc + ^ 



(47) 



We can derive the conditions necessary for the extremal P app (r), and we can represent them 
using the following nonlinear simultaneous equations. 

-i 



1 




G 



k 



E- 

k PG k + h + 

El I 2 Gu 
\T k \ * 



if 



(48) 



-E 

27V ^ 



G 



k 



1 V ,^|2 (/3% + Z*) 2 

AT 2-^i 



k {PG k + h + ± 



2 ' 



(49) 



1 

I 



E 



fc 1 + 



E 



t $ G k + h + h 
l 



+£hbf— ? 

fe 2f2 ( + h + i 



(50) 



We can then determine the convergence point by means of an iterative calculation. 

We next determined the unknown hyperparameter, f , by maximizing the marginal like- 
lihood, assuming that (3 = (3 and h — h. Thus, we restricted ourselves to the case where 
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the generation model of the image was already known, as was j3 and h. Curve (C) in Fig. 
[I] shows the mean squared error when the image was restored using the estimated f. Curve 
(C) in Fig. |2| shows the estimated f. 

Note that the difference between curve (B) in Fig. |2|, the limit of the restoration, and curve 

(C) , obtained by minimizing the mean squared error increased with the noise correlation pa- 
rameter, a. Therefore, the error obtained by maximizing the marginal likelihood (curve (C) 
in Fig. [I]) increases with the noise correlation parameter, a. This implies that without know- 
ing the noise model, estimating the hyperparameters by maximizing the marginal likelihood 
does not work well. That is, the conventional uncorrelated noise model cannot cope with 
spatially correlated noise. 

To confirm this, we used maximization of the marginal likelihood to estimate (3 and /i-the 
hyperparameters of the image-generation probability-as well as f. Curve (D) in Fig. [!] shows 
the mean squared error estimated using hyperparameters /?, h, and f. Curve (D) in Fig. |2| 
shows the obtained f, and Fig. [3] shows the obtained (3 and h. The error shown by curve 

(D) in Fig. [I] is much greater than that shown by curve (C). 
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FIG. 4: Mean squared error and marginal likelihood. Horizontal axis denotes hyperparameter 
f; left vertical axis denotes mean squared error (B) given by Eq.(|35|); right vertical axis denotes 
logarithmic marginal likelihood (C) given by Eq.(f47|). The parameters were a = 1.0, (5 = 1.0, 
h = 1.0, and k = 3.0. Those used for (B) correspond to those for curve (B) in Fig. [l]when a = 1.0. 
Likewise, those used for (C) correspond to those for curve (C) in Fig. 0(C) when a = 1.0. 

Next, we demonstrate that the logarithmic marginal likelihood, curve (C) in Fig. [I], does 
not have a local maximum and that the iteratively obtained solution of Eq. (fl9| ) corresponds 
to the global maximum. Curve (B) in Fig. |] shows the mean squared error, Ei, given by 
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Eg. (|3~5"D , and curve (C) shows the logarithmic marginal likelihood, \n(P app (T)) , given by Eq. 
(f!7|). The horizontal axis denotes f. Curve (B) shows that E\ takes a minimum value, 
Ei = 0.12, at f = 3.2. The parameters used for curve (B) correspond to those for curve (B) 
in Fig. [I] when a = 1.0. Curve (C) shows that that the logarithmic marginal likelihood takes 
a maximum value, E\ = 0.28, at f = 0.2. The parameters used for curve (C) correspond to 
those for curve (C) in Fig. |I] when a = 1.0. The important point is that the logarithmic 
marginal likelihood, curve (C) in Fig. |], does not have a local maximum. 



C. Sample of images 

Next, we show the practical significance of these differences for a typical set of artificial 
images. Figure |5|(a) shows an original image generated using hyperparameters j3 = 0.5 and 




(a) Original image 




(d) Image restored (e) Image restored 

by minimizing by maximizing 

mean squared error marginal likelihood 



FIG. 5: Artificial images. N = 64 2 . 
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h = l.O -4 . Figure [|(c) shows the same image after it was distorted by the noise shown 
in Fig. |5](b) (a = 1.0, b = 0.75, and k = 3.0). The mean squared error, E2, between the 
original and distorted images was 0.57. Figure ||(d) shows the image after it was restored 
using the complete restoration model: the generation model and noise model were consistent 
with the original models, and optimum values were used for the hyperparameters. When 
a = 1.0, curve (A) in Fig. [I] corresponds to Fig. |5](d), and the restored mean squared 
error, E±, is 0.27. Figure ||(e) shows the image after it was restored using an uncorrelated 
noise model with hyperparameters obtained by maximizing the marginal likelihood. When 
a = 1.0, curve (D) in Fig. [I] corresponds to Fig. ||](e), and the restored mean squared error 
is 0.45. The image in Fig. [|(d) resembles the original one, Fig. |^(a), while the one in Fig. 
|5](e) is similar to the distorted one in Fig. ^(c) and looks slightly out of focus. 

Furthermore, we show the results when natural images were used. Figure [| shows the 
natural images and a table of the estimated parameters. We used b = 0.75 and k = 3.0 for 
the noise parameters and three values for a: 0.0, 0.5, and 1.0. The three images on the left 
are distorted, and those on the right are the same images restored using /3, h, and f estimated 
using the maximized marginal likelihood. In this simulation as well, E\ increased with the 
noise correlation parameter, a. This tendency is reflected in the three restored images. For 
a = 1.0, the lower right image, the noise looks like a stain that was not completely removed. 

D. Discussion 

Here, we consider why line (A) in Fig. [I] tends to decrease as a approaches 1.0. This 
phenomenon appears to be universal, and also occurs with another parameter set as shown 
in Fig. |7|(ii). For reference, we also show line (i) which is based on the parameter set that 
was used in Fig. |l|(A) with only N changed to iV = 64 2 . (Note that line (A) is independent 
of N). 

For the original image, we used the Gaussian model represented by Eq. ([[]), which 
has neighboring interactions on each pixel. For the noise image, we used the Gaussian 
model represented by Eq. (H), in which the interaction is Gaussian functionally decreased. 
We expected the difference between these two models to become conspicuously large near 
a = 1.0, and this difference between the models would enable successful restoration near 
a = 1.0. Figure || shows the typical images about Fig. 0(ii). The distorted image was most 
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FIG. 6: Natural images. N = 64 2 . 

difficult to restore in Fig. |8| when a = 0.55, as shown in Fig. 0(ii). In this case, the original 
and the noise images were visually similar; more precisely, the correlation lengths of the two 
images were similar. Thus, to successfully restore a distorted image, it is desirable that the 
statistical properties of the original image should be different from those of the noise image. 

Here, we summarize our explanation of why line (A) in Fig. [I] decreases near a = 
1.0. As a approaches 1.0, the mean squared error decreases because the difference of the 



16 



0.6 
g 0.5 

1 

CD 

-O 0.4 
CD 

g 0.3 
CT 

£ 0.2 
03 

® 0.1 
0.0 



(ii) /3=0.5, h = W A , K=7.0 



(i) /3=1.0, ft = 1.0, K=3.0 



0.0 0.2 0.4 0.6 0.. 
a from Eq.(10) 



1.0 



FIG. 7: Mean squared error between original image and restored image. The horizontal axis 
denotes a from Eq. (|l0|), and the vertical axis denotes the mean squared error. As a criterion, the 
optimal decode given by Eq. ( |37| ) is used. The parameters b = 0.75, and N = 64 2 are fixed. For 
the solid line (i), the other parameters were set as (5 = 1.0, h = 1.0, and k = 3.0. For the dashed 
line (ii), the other parameters were set as (3 = 0.5, h = 10~ 4 , and k = 7.0. 

statistical properties, especially the correlation length, between the two images enables easier 
restoration. Conversely, the mean squared error is greater at a = 0.55 on line (ii) in Fig. [7] 
because the correlation-length similarity between the images makes it difficult to distinguish 
between them, and that also makes it difficult to restore the distorted image. To ensure 
successful restoration, the correlation length of the original image should be different from 
that of the noise image, which is the case when a = 0.00 and a = 1.00 in Fig. [8[ 



V. CONCLUSION 



We investigated the use of the Bayesian inference to restore images under conditions of 
spatially correlated noise. We assumed that both the original image and the noise obeyed a 
Gaussian distribution composed of translational symmetric matrices. We used the Fourier 
transformation to diagonalize the covariance matrices, which enabled us to apply various 
forms of statistical analysis. We obtained the expected value of a restored image and ob- 
tained the optimal analytical hyperparameters by minimizing the mean squared error and 
used these hyperparameters to determine the generation and noise models. Furthermore, we 
discussed whether the conventional spatially uncorrelated noise model could cope with the 
spatially correlated noise or not. We used two methods to estimate the hyperparameters: 
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Original image Noise image Restored image 



FIG. 8: Typical images about Fig. (ii). N = 64 2 . Parameters for the original images were 
(3 = 0.5 and h = 10~ 4 . The three original prepared images were the same. Parameters for noise 
images were (3 = 0.5, h = 10~ 4 ,k = 7.0, and b = 0.75. Only a was changed. Note that when 
a = 0.55, the original and noise images were visually similar, and the restoration quality was lower 
in this case. 

minimizing the mean squared error and maximizing the marginal likelihood. The difference 
between the errors obtained using the two methods increased with the noise correlation 
parameter. The restoration error was larger when we used the hyperparameters obtained 
using the maximization of the marginal likelihood method. Thus, the conventional spatially 
uncorrelated noise model could not cope with the spatially correlated noise. 
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